This invention relates to augmented virtual retinal display devices, and more particularly to a system for tracking viewer position and for adding data to a view based upon viewer position.
A virtual retinal display device is an optical device for generating an image upon the retina of an eye. Light is emitted from a light source, passed through a lens, then deflected along a scan path by a scanning device. At a distance defined by the lens the scanned light converges to a focal point for each pixel position. As the scanning occurs the focal point moves along to define an intermediate image plane. The light then diverges beyond the plane. An eyepiece is positioned along the light path beyond the intermediate image plane at some desired focal length. An xe2x80x9cexit pupilxe2x80x9d occurs shortly beyond the eyepiece in an area where a viewer""s eye pupil is to be positioned.
A viewer looks into the eyepiece to view an image. The eyepiece receives light that is being deflected along a raster pattern. Light thus impinges on the viewer""s eye pupil at differing angles at different times during the scanning cycle. This range of angles determines the size of the image perceived by the viewer. Modulation of the light during the scanning cycle determines the content of the image.
An augmented virtual retinal display is a see-through display which overlays an image onto a background. The overlaid image is a virtual image. The background is a real world view of the ambient environment. The overall image is formed by adding light to the background. The added light corresponds to the virtual image. The virtual image appears to be transparent because in the display portion at which the image is formed, light from both the virtual image and the background impinge on the same photoreceptors in the viewer""s eye.
According to the invention, a scanned beam tracking system is included in a virtual retinal display. The function of the tracking system is to provide information for determining where a user is looking. In a preferred embodiment head position and orientation is tracked. Information is displayed based upon the viewer""s eye position.
According to one aspect of the invention, a non-visible light source (e.g., infrared light source) generates light for scanning the viewer""s environment in the direction the viewer""s head is looking. A visible light source generates visible light which is scanned on a viewer""s retina to generate a virtual image. A common scanning system is used to scan both the non-visible light and the visible light. The visible light is directed into the viewer""s eye. The non-visible light is directed away from the viewer""s eye into the environment. Thus, the scanning rate for the tracking system is the same as the scanning rate for the virtual display.
According to another aspect of the invention, a beamsplitter with an infrared mirror reflects the infrared light away from the viewer""s eye toward the environment, while passing visible light (e.g., virtual image and background light from environment) toward from the viewer""s eye.
According to another aspect of the invention, infrared reflectors are positioned in the environment. When the infrared light from the virtual retinal display scans over a reflector the reflector directs the infrared light back toward the virtual retinal display. The virtual retinal display beamsplitter with infrared mirror deflects such light away from the viewer""s eye along a path toward an infrared detector. The round trip time of the infrared light is substantially less than the scanning time for scanning an image frame onto the viewer""s retina. Thus, the position of the reflector is known relative to the scanning cycle. Specifically, the current pixel of the scanning cycle when the infrared return light is detected corresponds to the position of the reflector.
According to another aspect of this invention, multiple reflectors are positioned in the environment. In some embodiments, a reflector has a reflection pattern identification which allows the system to know which reflector or which type of reflector is being scanned at a given time during the scan cycle.
According to another aspect of this invention, when at least three reflectors are scanned during a given scan cycle, the system can triangulate a precise position of the user relative to such reflectors.
According to another aspect of the invention, an image, graphic information or text information is added to the display imagery when a reflector is scanned. In one embodiment, such added information is stabilized relative to the head position. For example, such information is always displayed at a prescribed portion of the display (e.g., upper right portion) when a reflector is scanned. In another embodiment such information is fixed relative to the background environment. In an exemplary embodiment the reflector is placed upon a target object. When the reflector is detected, the target object is within the field of view of the user. Textual information about the target object is displayed in a prescribed portion of the field of view (e.g., lower right corner). Even when the user""s head moves, the textual information stays fixed in the prescribed portion of the field of view as long as the target object remains within the field of view. Once the user looks in a direction which excludes the target object from the field of view, the textual information is removed. Thus, the added information is stabilized relative to the head.
In another embodiment the added information is stabilized relative to the background. For example, a predetermined virtual image is overlaid onto the background at a position registered to the background (e.g., a virtual image ornament is displayed to appear on a given branch of a real tree within the real background viewed by the user). Even when the viewer""s head moves (and thus the virtual retinal display), as long as the desired location is still within view of the user, then the information is added to the display at a point fixed relative to the background (e.g., the virtual ornament appears at the same spot on the real tree).
According to another aspect of the invention, the working volume for the tracking system corresponds to the field of view of the retinal display. Thus, any reflectors within the field of view are detected.
According to another aspect of the invention, an augmented virtual retinal display system with view tracking receives an image data signal for generating a virtual image upon a viewers eye. The system also receives background light from a real environment for passing a real environment background image to the viewer""s eye. The system has a field of view for viewing the real environment and the virtual image. The system includes a light source for generating visible light and infrared light. A modulator modulates the visible light as a function of the image data signal to define a sequence of display pixels forming the virtual image. A scanner receives the infrared light and the modulated visible light, and deflects the received visible light and infrared light along a raster pattern. The scanned visible light and infrared light impinge on a beamsplitter. The beamsplitter includes a coating at the incident surface which is reflective to the infrared light. The infrared light is reflected off the infrared reflective coating into the real environment. The scanned visible light passes through the infrared reflective coating then is in-part reflected and in-part passed. The reflected portion of the visible light enters the real environment. The passed portion impinges on a concave mirror, then is reflected back to the beamsplitter, and in turn, deflected toward the viewer""s eye. Also impinging on the infrared reflective portion of the beamsplitter is the background light and the returning infrared light reflected from the real environment. Background light passes through the beamsplitter and travels a path destined for the viewer""s eye. An infrared reflector is located in the real environment. Scanned infrared light from the display enters the real environment, (e.g., along the raster scanning pattern). Some of such infrared light is reflected back to the display. More specifically, when the infrared reflector is within a field of view of the scanning pattern, the infrared light impinges on the reflector and is reflected back onto the infrared reflective coating of the beamsplitter. The coating reflects the re-entering infrared light along a path within the display destined for an infrared detector. The infrared detector generates a first signal in response to detection of the re-entering infrared light.
According to another aspect of the invention, the infrared reflector includes a center area reflective to infrared light and a plurality of spaced concentric rings reflective to infrared light.
According to another aspect of the invention, the system is used in combination with a processing system. The processing system generates the image data signal and receives the first signal. The processor correlates the detection of the re-entering infrared light to a display pixel among the sequence of display pixels forming the virtual image. The processing system also embeds display data into the image data signal responsive to the received first signal. The embedded display data corresponds to virtual imagery fixed relative to the field of view and/or to virtual imagery fixed relative to the infrared reflector in the environment.
According to another aspect of the invention, a method for overlaying a virtual image onto a real background image with an augmented virtual retinal display having view tracking is provided. The augmented virtual retinal display apparatus receives an image data signal for generating pixels of the virtual image upon a viewer""s eye and receives background light from a real environment for passing the real background image to the viewer""s eye. The method includes generating visible light and infrared light, and modulating the visible light as a function of the image data signal to define a sequence of display pixels forming the virtual image. At another step the received visible light and infrared light are deflected along a raster pattern. At another step the deflecting infrared light is received at an infrared mirror and deflected into the real environment. At another step the deflecting visible light is received at a beamsplitter and deflected the visible light toward a viewer""s eye. At another step the background light is received at the beamsplitter. A portion of the background light is passed toward the viewer""s eye. At another step the infrared light deflected into the real environment is reflected by an infrared reflector located in the real environment back to the infrared mirror as re-entering infrared light. At another step the re-entering infrared light is detected at an infrared detector. At another step a pixel of the virtual image is correlated with the detected re-entering infrared light.
According to one advantage of this invention, there is a latency of less than one display frame period between when a reflector is detected and when information responsive to such detection is added to the display. According to another advantage of the invention, the positioning of information added to the display as function of reflector detection is accurate to within one display pixel. These and other aspects and advantages of the invention will be better understood by reference to the following detailed description taken in conjunction with the accompanying drawings.